Welcome to Kafka Streaming Process. After watching this video, you will be able
to: Describe what the Kafka Streams API is and
its main benefits. Describe what the Kafka Stream processing
topology is. In event streaming, in addition to transporting
data, data engineers also need to process data through, for example, data filtering,
aggregation, and enhancement. Any applications developed to process streams
are called stream processing applications. For stream processing applications based on
Kafka, a straightforward way is to implement an ad hoc data processor to read events from
one topic, process them, and publish them to another topic. Let’s look at an example. You first request raw weather JSON data from
a weather API, and you start a weather producer to publish
the raw data into a raw weather topic. Then you start a consumer to read the raw
weather data from the weather topic. Next, you create an ad hoc data processor
to filter the raw weather data to only include extreme weather events, such as very high
temperatures. Such a processor could be a simple script
file or an application which works with Kafka clients to read and write data from Kafka. Afterwards, the processor sends the processed
data to another producer, and it gets published to a processed weather topic. And finally, the processed weather data will
be consumed by a dedicated consumer and sent to a dashboard for visualization. Such ad hoc processors may become complicated
if you have many different topics to be processed. A solution that may solve these challenges
is Kafka. It provides the Streams API to facilitate
stream processing. Kafka Streams API is a simple client library
aiming to facilitate data processing in event-streaming pipelines. It processes and analyzes data stored in Kafka
topics, thus both the input and output of the Streams API are Kafka topics. Additionally, Kafka Streams API ensures that
each record will only be processed once. And finally, it processes only one record
at a time. Kafka Streams API is based on a computational
graph called a stream-processing topology. In this topology, each node is a stream processor
which receives streams from its upstream processor, performs data transformations such as mapping,
filtering, formatting, aggregation, and produces output streams to its downstream stream processors. Thus, the edges of the graph are I/O streams. There are two special types of processors: On the left, you can see the source processor
which has no upstream processors. A source processor acts like a consumer which
consumes streams from Kafka topics and forwards the processed streams to its downstream processors. On the right, you can see the sink processor,
which has no downstream processors. A sink processor acts like a producer which
publishes the received stream to a Kafka topic. Let’s re-design the previous weather stream
processing application with Kafka Streams API. Suppose you have a raw weather topic and a
processed weather topic in Kafka. Now, instead of spending a huge amount of
effort developing an ad hoc processor, you could just plug in the Kafka Streams API here. In the Kafka Streams topology, we have three
stream processors: The source processor that consumes raw weather
streams from the raw weather topic and forwards the weather stream to the stream processor to filter the stream
based on high temperature. Then, the filtered stream will be forwarded
to the sink processor which then publishes the output to the processed weather topic. Concluding, this is a much simpler design
than an ad hoc data processor, especially if you have many different topics to be processed. In this video, you learned that: Kafka Streams API is a simple client library
to help data engineers with data processing in event streaming pipelines. A stream processor receives, transforms, and
forwards the streams. Kafka Streams API is based on a computational
graph called a stream processing topology. And in the topology, each node is a stream
processor, while edges are the I/O streams. And finally, in this topology, we find two special types
of processors: The source processor and the sink processor.